Prediction method for healthy radius of blood vessel path, prediction method for candidate stenosis of blood vessel path, and blood vessel stenosis degree prediction device

ABSTRACT

A computer-implemented method for predicting a blood vessel stenosis is disclosed. The method may include extracting a blood vessel path and its centerline based on the image of the blood vessel. The method may further include determining a candidate stenosis for the blood vessel path and identifying image blocks along the centerline of the blood vessel path within a range of candidate stenosis for the blood vessel path determined based on the candidate stenosis. The method may also include determining a degree of stenosis for the blood vessel path by applying a trained learning network comprising a convolutional neural network and a recurrent neural network on the image blocks within the range of candidate stenosis.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.16/580,981, filed Sep. 24, 2019, which claims the benefit of priority toU.S. Provisional Application No. 62/735,829, filed on Sep. 24, 2018, andChinese Patent Application No. 2019102628380, filed on Apr. 2, 2019. Theentire contents of all these priority applications are incorporatedherein by reference.

TECHNICAL FIELD

The present disclosure relates to methods and devices for predictingblood vessel physiological parameters, for example to a method forpredicting a healthy radius of a blood vessel path (a segment of a bloodvessel), a method for predicting candidate stenosis of a blood vesselpath, and a device for predicting the degree of blood vessel stenosisusing machine learning.

BACKGROUND

Recent studies have shown that degree of blood vessel stenosis is ofsignificance for the diagnosis and the prevention of cardiovasculardisease. Generally, the degree of blood vessel stenosis is manuallydetermined: the expert judges the degree of blood vessel stenosis bymanual observation of coronary CT angiography (CTA) and/or quantitativecoronary angiography (QCA). However, such judgment is time consuming,tedious, wastes huge amounts of resources, and relies heavily on theexpert's experience.

Although some methods for automatically or semi-automaticallydetermining the degree of vascular stenosis have been proposed, thesemethods either manually determine a detection area of degree of vascularstenosis by an expert or perform a global scan of a large section ofblood vessels, wherein the former ones still rely heavily on expert'sexperience and the latter ones result in a large amount of computationalresources being consumed and a slow computation speed, which does notsatisfactorily meet clinical time requirements.

Traditional methods for automatically or semi-automatically determiningblood vessel stenosis have also introduced machine learning, but thesemachine learning-based stenosis detection systems typically includemultiple modules, such as a feature extraction module, a stenosisdetection module and a stenosis weight estimation module. Generally, thefeature extraction module manually specifies a feature extraction mode,which cannot be adaptively adjusted for different application scenarios.The stenosis detection module and the stenosis weight estimation moduleare independently trained, and the objective functions are different.The independent training of the modules requires corresponding trainingdata, which has higher requirements for training data: the trainingcannot be complemented by cross-optimizing during the training process;and the optimal performance of the overall system cannot be achieved.

The present application has been proposed to solve the above-identifiedproblems as well as for other purposes.

SUMMARY

Certain embodiments may provide an automated prediction method for thehealthy radius of the blood vessel path implemented by a computer. Themethod does not rely on expert's experience and the predicted healthyradius agrees well with the actually measured healthy radius of theblood vessel path. Certain embodiments may provide a prediction methodfor the candidate stenosis of the blood vessel path implemented by acomputer. The method does not rely on expert's experience toautomatically and accurately determine candidate stenosis on the bloodvessel path that requires further medical intervention. Certainembodiments may also or alternatively provide a blood vessel stenosisdegree prediction device capable of automatically determining acandidate stenosis of the blood vessel path and detecting the degree ofstenosis for candidate stenosis range of the blood vessel path, whichcan significantly reduce computation load and improve detectionefficiency. At the same time, certain embodiments can effectively avoidmissed detections. The device can realize the end-to-end detection ofthe stenosis from the image block sequence to the corresponding positionof the entire blood vessel path by using the learning network. Thetraining of the learning network may also be end-to-end mode. Trainingdata sets may be easier to obtain and may enable a well-trained learningnetwork to achieve good predictive performance of the device as a whole.

According to a first aspect of the present disclosure, there is provideda prediction method for a healthy radius of a blood vessel path, whichis implemented by a computer, the predicting method including the stepsof: acquiring a blood vessel radius of the blood vessel path; by aprocessor, detecting radius peak of the blood vessel radius of the bloodvessel path; and by the processor, predicting the healthy radius of theblood vessel on the blood vessel path by performing regression on theradius peak of the blood vessel radius.

In some embodiments, the step of predicting the healthy radius of theblood vessel on the blood vessel path by performing regression on theradius peak of the blood vessel radius includes: predicting a referencehealthy radius of the blood vessel on the blood vessel path byperforming linear regression on the radius peak of the blood vesselradius; processing the radius peak in the blood vessel radius based onpredicted reference healthy radius of the blood vessel on the bloodvessel path; and predicting the healthy radius of the blood vessel onthe blood vessel path by performing a quadratic regression on theprocessed radius peak in the vessel radius.

In some embodiments, the step of processing the radius peak in the bloodvessel radius based on predicted reference healthy radius of the bloodvessel on the blood vessel path includes: replacing the radius peakamong the radius peaks of the blood vessel radius that is lower than thecorresponding reference healthy radius with the corresponding referencehealthy radius.

In some embodiments, the regression includes a Gaussian processregression.

According to a second aspect of the present disclosure, there isprovided a prediction method for a candidate stenosis of a blood vesselpath, which is implemented by a computer, the prediction methodincluding: a prediction method for a healthy radius of a blood vesselpath according to various embodiments of the present disclosure; by aprocessor, detecting a radius valley of a blood vessel radius on theblood vessel path; and by a processor, determining the candidatestenosis based on the radius valley and the healthy radius of the bloodvessel path.

In some embodiments, the step of determining the candidate stenosisbased on the radius valley and the healthy radius of the blood vesselpath includes: determining a ratio of the radius valley to the healthyradius throughout the blood vessel path, and determining it as thecandidate stenosis where the ratio is less than a first predeterminedthreshold; or determining a ratio of a difference between the healthyradius and a radius valley to a healthy radius throughout the bloodvessel path, and determining it as the candidate stenosis where theratio is greater than a second predetermined threshold.

In some embodiments, the regression includes a Gaussian processregression.

According to a third aspect of the present disclosure, a blood vesselstenosis degree prediction device is provided, the blood vessel stenosisdegree prediction device includes: an interface configured to receive animage of a blood vessel; a memory storing executable instructions; aprocessor configured to implement following steps by executing theexecutable instructions: extracting a blood vessel path and itscenterline based on an image of the blood vessel; determining candidatestenosis for each blood vessel path; setting a range of the candidatestenosis for each blood vessel path based on the determined candidatestenosis; obtaining image blocks along the centerline within the rangeof candidate stenosis for each of the blood vessel path; and based onthe obtained image blocks, determining the degree of stenosis for eachblood vessel path by using a trained learning network composed of aconvolutional neural network and a recurrent neural network.

In some embodiments, the range of candidate stenosis for each bloodvessel path is a length centered on the determined candidate stenosis.

In some embodiments, the candidate stenosis of each blood vessel path isdetermined by: obtaining a blood vessel radius of the blood vessel path;detecting a radius peak and a radius valley in a blood vessel radius ofthe blood vessel path; predicting reference healthy radius of the bloodvessel path by performing linear regression on the radius peak in theblood vessel radius: replacing the radius peak among the radius peaks inthe blood vessel radius that is lower than the corresponding referencehealthy radius with the corresponding reference healthy radius;predicting the healthy radius of the blood vessel path by performing aquadratic regression on the replaced radius peak in the blood vesselradius; and determining the candidate stenosis based on the radiusvalley and the healthy radius of the blood vessel path.

In some embodiments, the step of determining the candidate stenosisbased on the radius valley and the healthy radius of the blood vesselpath includes: determining a ratio of the radius valley to the healthyradius throughout the blood vessel path, and determining it as thecandidate stenosis where the ratio is less than a first predeterminedthreshold; or determining a ratio of a difference between the healthyradius and a radius valley to a healthy radius throughout the bloodvessel path, and determining it as the candidate stenosis where theratio is greater than a second predetermined threshold.

In some embodiments, the recurrent neural network is a bidirectionalrecurrent neural network.

In some embodiments, the blood vessel stenosis degree predicting devicefurther includes: an output unit that is configured to output at leastone of the stenosis, the range of stenosis, and the stenosis value,which may be a numeric representation of the degree of stenosis, of eachblood vessel path.

According to a fourth aspect of the present disclosure, there isprovided a non-transitory storage medium having stored thereon computerexecutable instructions that, when executed by a processor, implement atleast one of the following methods: the prediction method for a healthyradius of a blood vessel path according to various embodiments of thepresent disclosure; the prediction method for a candidate stenosis of ablood vessel path according to various embodiments of the presentdisclosure; and a method for predicting a degree of stenosis of a bloodvessel, the method including the steps of: extracting a blood vesselpath and its centerline based on an image of the blood vessel;determining candidate stenosis for each blood vessel path; setting arange of the candidate stenosis for each blood vessel path based on thedetermined candidate stenosis; obtaining image blocks along thecenterline within the range of the candidate stenosis for each bloodvessel path; and based on the obtained image blocks, determining thedegree of stenosis for each blood vessel path by using a trainedlearning network composed of a convolutional neural network and arecurrent neural network.

According to the prediction method for the healthy radius of the bloodvessel path according to various embodiments of the present disclosure,it is possible to predict the healthy radius of the blood vessel paththat is in good agreement with the actual situation. The predictionmethod for the candidate stenosis of the blood vessel path and the bloodvessel stenosis degree prediction device according to variousembodiments of the present disclosure can automatically determine thecandidate stenosis and detect the degree of stenosis for the candidatestenosis range, which can significantly reduce the computation load,improve the detection efficiency, and effectively avoid misseddetection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, likereference numerals may describe similar components in different views.Like reference numerals having letter suffixes or different lettersuffixes may represent different instances of similar components. Thedrawings illustrate generally, by way of example, but not by way oflimitation, various embodiments, and together with the description andclaims, serve to explain the disclosed embodiments.

FIG. 1 illustrates a flow chart of a method for predicting a healthyradius of a blood vessel path in accordance with an embodiment of thepresent disclosure.

FIG. 2(a) shows a comparison of a healthy radius of a blood vesseldiameter predicted by a first prediction method and actually measuredhealthy radius and lesion radius, in accordance with an embodiment ofthe present disclosure.

FIG. 2(b) shows a comparison of a healthy radius of the vessel diameterpredicted by a second prediction method and the actually measuredhealthy radius and lesion radius, in accordance with another embodimentof the present disclosure.

FIG. 2(c) shows a comparison of a healthy radius of the vessel diameterpredicted by a third prediction method and actually measured healthyradius and lesion radius, in accordance with yet another embodiment ofthe present disclosure.

FIG. 3 illustrates a schematic diagram of a process for predicting adegree of stenosis of a blood vessel according to an embodiment of thepresent disclosure.

FIG. 4 illustrates a structural diagram of a learning network forpredicting a degree of stenosis of a blood vessel according to anembodiment of the present disclosure.

FIG. 5 illustrates a flow diagram for training a learning network forpredicting a degree of stenosis of blood vessels in accordance with anembodiment of the present disclosure.

FIG. 6 shows a schematic illustration of a system for predicting adegree of stenosis of blood vessel in accordance with an embodiment ofthe present disclosure.

FIG. 7 shows a block diagram of a device for predicting a degree ofstenosis of blood vessel according to an embodiment of the presentdisclosure.

DETAILED DESCRIPTION

The following technical terms have a uniform meaning in the presentdisclosure. The technical term “blood vessel path” refers to a path of ablood vessel from an inlet to an outlet; for example, a vessel tree mayinclude multiple blood vessel paths. The technical term “blood vesselradius of a blood vessel path” means the radius of a blood vessel thatis present throughout the blood vessel path, that is, a series of bloodvessel radii corresponding to the radius of the blood vessel. Thetechnical term “radius peak in the blood vessel radius of a blood vesselpath” refers to a sequence of radius peaks in a series of vessel radiithat are present throughout the blood vessel path.

FIG. 1 shows an illustration of a process 100 of a method for predictinga healthy radius of a blood vessel path, in accordance with anembodiment of the present disclosure. As shown in FIG. 1, the predictionprocess 100 may start from acquiring a vessel radius of a blood vesselpath (step 101). This step 101 can be implemented in various ways, forexample, a device implementing the process can receive a medical imageof a blood vessel collected by an image acquisition device, which can bea CT image taken along the longitudinal direction of the blood vessel,or digital subtraction angiography images at a number of differentprojection angles for the same blood vessel, etc. Then, based on themedical image of the collected blood vessel, various open source toolssuch as VMTK may be used to extract the center line and the blood vesselwall of the blood vessel path, thereby reconstructing athree-dimensional geometric model of the blood vessel, and extractingthe radius of the blood vessel throughout the blood vessel path from thereconstructed three-dimensional geometric model, such as, but notlimited to, the radius of the blood vessel throughout the centerline ofthe blood vessel path. Next, a processor may be used to detect a radiuspeak in a blood vessel radius of the blood vessel path (step 102).Specifically, among a series of blood vessel radii throughout the bloodvessel path, a part of which is a local radius peak, and these localradius peaks are detected and arranged in order. Regression (step 103)may be performed on the sequence of these local radii peaks sequentiallyarranged by the processor to predict the healthy radius of the bloodvessel path. This method for predicting the healthy radius can beautomatically implemented, and the medical image of the blood vessel canbe directly used as an input, which is friendly to the user and has afast computation speed, and is in some extent compatible with thehealthy radius distribution of the clinically detected blood vesselpath.

In various embodiments in accordance with the present disclosure, theprocessor may be a processing device including one or more generalpurpose processing devices, such as a microprocessor, central processingunit (CPU), graphics processing unit (GPU), and the like. Morespecifically, the processor may be a Complex Instruction Set Computing(CISC) microprocessor, a Reduced instruction Set Computing (RISC)microprocessor, a Very Long Instruction Word (VLIW) microprocessor, aprocessor running other instruction sets or a processor that runs acombination of instruction sets. The processor may also be one or morededicated processing devices such as an application specific integratedcircuit (ASIC), field programmable gate array (FPGA), digital signalprocessor (DSP), system-on-chip (SoC), and the like. As will beappreciated by those skilled in the art, in some embodiments, theprocessor may be a dedicated processor rather than a general purposeprocessor. The processor may include one or more known processingdevices such as a Pentium™, Core™, Xeon™ or Itanium series ofmicroprocessors manufactured by Intel™, Turion™, Athlon™, Sempron™,Opteron™, FX™, Phenom™ series manufactured by AMD™, or variousprocessors manufactured by Sun Microsystems. The processor may alsoinclude a graphics processing unit such as a GPU from GeForce®, Quadro®manufactured by Nvidia™, Tesla® series, GMA manufactured by Intel™,Iris™ series, or Radeon™ series manufactured by AMD™. The processor mayalso include an accelerated processing unit such as the Desktop A-4 (6,6) series manufactured by AMD™ the Xeon Phi™ series manufactured byIntel™, and the like.

In some embodiments, regression may be implemented in various ways instep 103, including but not limited to spline fitting. Bayesian linearregression, piecewise binomial fitting, and the like. For example,Gaussian process regression can be used, and a kernel function may beintroduced in the Gaussian process regression, so that the regressioneffect for the frequently fluctuated radius distribution curve of theblood vessel path is significantly better than other regression methods.

The various implementations of step 103 will be described below bytaking Gaussian process regression as an example.

In some embodiments, a linear Gaussian process regression can bedirectly performed with the radius peak of the blood vessel path topredict the healthy radius of the blood vessel path, and the predictedhealthy radius of the blood vessel path is shown by the solid line inFIG. 2(a). In FIG. 2(a), the distribution curve of the lesion radius ofthe blood vessel path is shown by a broken line, and the healthy radiusactually measured when the lesion is not present in the current severelystenosis segment is shown by a bold line. It can be seen that thehealthy radius of the predicted blood vessel path is relatively lowcompared with the actually measured healthy radius.

In some embodiments, a linear regression can also be performed on thepeak of the radius in the blood vessel radius to predict the referencehealthy radius of the vessel on the blood vessel path, and based on thereference healthy radius, the radius peak in the blood vessel radius canbe processed. For example, a radius peak below the correspondingreference healthy radius is removed from the radius peak in the bloodvessel radius (hereinafter referred to as “removal processing” forshort), or the radius peak below the reference healthy radius isreplaced with the corresponding reference healthy radius (hereinafterreferred to as “replacement processing” for short). The healthy radiusof the blood vessel on the blood vessel path is then predicted byperforming a second regression on the radius peak in the processed bloodvessel radius. The solid line in FIG. 2(b) shows the healthy radius ofthe blood vessel path predicted by the “removal process”, and the solidline in FIG. 2(c) shows the healthy radius of the blood vessel pathpredicted by the “replacement process”. As shown, the former is higherthan the actually measured healthy radius, the latter is moderate, looksmore reasonable, and is in good agreement with the actual measuredhealthy radius distribution. Using the healthy radius of the bloodvessel path predicted by the “removal process”, two candidate stenoses(e.g. a stenosis with a degree of stenosis greater than 0.2) can bedetected, and the degree of stenosis is a ratio of the differencebetween magnitude of the valley at the stenosis and the predictedhealthy radius of the corresponding position to the predicted healthyradius of the corresponding position. The degrees of stenosis of the twostenosis are 0.22 and 0.75, respectively (see FIG. 2(b)). Using thehealthy radius of the blood vessel path predicted by the “replacementtreatment”, a candidate stenosis can be detected, where the degree ofstenosis of the stenosis is 0.74 (see FIG. 2(c)), and the doctorsmanually measured the same blood vessel. The results (the actual numberof stenosis and the actual measured degree of stenosis) are moreconsistent with the latter.

Although the healthy radius of the blood vessel path may not directlyyield medical diagnosis results, the healthy radius of the blood vesselpath can serve as an intermediate parameter that plays an important rolein diagnosis. For example, the healthy radius of the blood vessel pathcan be used to further detect stenosis in the blood vessel path, and canalso be used to further detect the abnormal bumps in the blood vesselpath and so on. In certain embodiments, these further detection stepsare also performed by a processor of the device implementing the method.Additionally, yet further steps such as stenosis removal steps using,for example, angioplasty, may likewise be performed by a device usinginput provided from the preceding steps. Other computer-implementedmedical intervention based on the preceding prediction and detectionsteps are also permitted.

As used herein, the technical term “candidate stenosis” refers to aplace where a significant stenosis requiring medical intervention islikely to occur. For example, the likelihood of the significant stenosisoccurring at the place may be greater than a predetermined threshold,such as 50%, 95%, or 99%. In some embodiments, a method for predicting acandidate stenosis of a blood vessel path is provided, the predictivemethod can employ a method for predicting a healthy radius of a bloodvessel path in accordance with various embodiments of the presentdisclosure to predict the healthy radius of the blood vessel path, andthe processor detects (a sequence of) the radius valley in the bloodvessel radius of the blood vessel path acquired in step 101, and thendetermines the candidate stenosis based on the detected radius of theblood vessel path and the healthy radius.

Candidate stenosis can be determined in various ways based on the radiusvalley and the healthy radius of the detected blood vessel path. Forexample, assuming that n valleys are detected on the blood vessel path(n is a natural number), the radius at each valley is r_(i) and thecorresponding healthy radius is h_(i) (i takes any natural numberbetween 1 and n), the first ratio r_(i)/h_(i) of the radius valley r_(i)to the corresponding healthy radius h_(i) can be determined, and if thefirst ratio r_(i)/h_(i) is less than a first predetermined threshold(e.g., 0.8), it may be determined here as the candidate stenosis. Insome embodiments, a second ratio (h_(i)−r_(i))/h_(i) may also bedetermined, and if the second ratio is greater than a secondpredetermined threshold (e.g., 0.2), then it may be determined here as acandidate stenosis.

FIG. 3 shows a schematic diagram of a process 300 for predicting thedegree of vessel stenosis in accordance with an embodiment of thepresent disclosure. The prediction process 300 may begin with receivinga medical image (not shown) of a blood vessel such as a blood vesseltree, which may be, for example, a CT image of a blood vessel tree,exemplified below by taking a CT image of a blood vessel tree as anexample. The blood vessel path and its centerline may be extracted basedon a CT image of the blood vessel tree using open source software suchas VMTK or using existing vessel reconstruction software (step 301). Theblood vessel tree may include multiple branches and thus may alsoinclude the blood vessel tree vessel path, and each blood vessel pathhas a corresponding centerline. In some embodiments, a 3D geometricmodel of each blood vessel path can be reconstructed by extracting acenterline and a vessel wall, and a sequence of vessel radii along thecenterline is extracted from the geometric model. In some embodiments,the extraction of the vessel radius along the centerline does notnecessarily reconstruct a 3D geometric model of the blood vessel path;for example, the blood vessel radius may also be determined from CTimages along the centerline, such as via image analysis.

At step 302, the various stenosis of each blood vessel path can bedetermined using various methods described above. For example, for eachblood vessel path, the candidate stenosis can be determined based on thesequence of blood vessel radii along its centerline. At step 303, therange of the candidate stenosis for each blood vessel path may be setbased on the determined candidate stenosis. For example, the range ofcandidate stenosis can be set to a length centered at the determinedcandidate stenosis. In some embodiments, the range of candidate stenosismay be set to a length that is centered at the candidate stenosis pointand that includes all points on both sides where the blood vessel radiir_(i) is less than the corresponding healthy radius h_(i).

Next, image units along the centerline within the candidate stenosisrange of each blood vessel path can be acquired (step 304). These imageunits along the centerline can reflect the spatial geometry of the bloodvessel path, for example, can be an image block taken along thecenterline, or in a 3D model block sampled in a 3D model of thereconstructed vessel tree along the centerline.

In step 305, image units along the centerline (such as but not limitedto image blocks) within the range of candidate stenosis of the acquiredindividual blood vessel paths may be taken as an input of the model andfed into the trained learning network to determine the stenosis of thecorresponding blood vessel path. By predicting the degree of stenosisfor the range of candidate stenosis for the blood vessel path, thesearch space can be significantly reduced, the prediction efficiency canbe improved, the computational load can be reduced, and the computationspeed can be increased. Moreover, taking advantage of the predictedhealthy radius that is highly consistent with the actual measured value,the range of candidate stenosis in which significant stenosis may occurcan be extracted from the entire blood vessel path, thereby ensuring thedetection rate and reducing the missed detection rate. The setting ofthe range of candidate stenosis and the prediction of the degree ofstenosis within the range can be automatically realized with the manualintervention significantly reduced. In practice, the labor burden of thehospital can be alleviated, and the dependence on the doctor'sexperience may be significantly reduced, which may be beneficial torapid and wide clinical implementation.

FIG. 4 illustrates a structural diagram of a learning network forpredicting a degree of stenosis of a blood vessel according to anembodiment of the present disclosure. As shown in FIG. 4, the learningnetwork can be constructed based on a convolutional neural network (CNN)and a recurrent neural network (RNN). An example is taken using an imageblock along a centerline as a model input. The learning network is anend-to-end learning model from an image block along the centerline to astenosis along the centerline. An image block along the centerline(x_(t), t may take any natural number between 1 and the total number ofsample points T) may be fed as a model input 401 to a corresponding CNNunit of the CNN layer 402, and the CNN unit acts as an encoder to learnand encode the image block x_(t) to learn local and spatial informationand produce a vector z_(t)=V(x_(t)) of fixed length. The vector z_(t) isthen fed into the recurrent neural network. In some embodiments, therecurrent neural network can include a forward RNN layer 404, a reverseRNN layer 403, an average pooling layer 405, and a regression layer 406.The vector z can be fed into the bidirectional RNN layer including theforward RNN layer 404 and the reverse RNN layer 403 to simultaneouslylearn a correlation in the key positive and negative directions of thesequence data (for the points on the blood vessel path, between theupstream and downstream points) using the forward RNN layer 404 and thereverse RNN layer 403. The forward RNN layer 404 and the reverse RNNlayer 403 of the bidirectional RNN layer are not connected by edges, andcan be separately trained by the general RNN training method, and theircomputations can be processed in parallel, which helps to improvecomputational efficiency. Subsequently, it is fed into the averagepooling layer 405 for up-sampling to extend the field of view of theprediction model to yield a more robust result. Here, the averagepooling layer is used as an example, but other up-sampling layers, suchas a maximum pooling layer, etc., may also be employed. The up-sampleddata using the average pooling layer 405 is fed to the regression layer406 to ultimately predict the degree of stenosis at various positions ofthe blood vessel path.

In FIG. 4, hatching is used to identify each node in the RNN layer.According to different requirements, each node can use long-term andshort-term memory (LSTM) recurrent neural network, gated recursive unit(GRU), bidirectional LSTM recurrent neural network, and bidirectionalGRU and the like. The bidirectional RNN can take into account therelationship between the detection point and its upstream and downstreamdetection points, and can improve the prediction accuracy of the degreeof stenosis.

FIG. 5 shows an illustration of a training process 500 for training alearning network for predicting degree of stenosis of blood vessels inaccordance with an embodiment of the present disclosure. The trainingprocess begins with receiving a training data set (step 501), taking animage block along the centerline as a model input, and each pair of datain the training data set is an image block

x₁, x₂, . . . , x_(T)

taken along T sampling points along the centerline and the artificiallylabeled degree of stenosis

y₁, y₂, . . . , y_(T)

(ground truth) at the corresponding position. At step 502, the receivedtraining data set may be loaded (e.g., all loaded or loaded in batch) ascurrent training data (step 502).

The parameters of the predictive network may be determined based on thetraining data (step 503) and verified against an objective function(step 504) to optimize the parameters for the training data. In someembodiments, the parameters of the predictive network may include atleast one of the number of network layers in the CNN portion, the numberof nodes per layer network, the number of hidden layers in the RNNportion, a learning rate, and an initial value. The optimal value ofthese parameters can be determined by cross-validation. As mentionedabove, the predictive network can be constructed by connecting the CNNencoder to the corresponding node of the bidirectional RNN. Thus, thepredictive network may contain parameters (V, W) where the parameter Vis for the CNN encoder portion and the parameter W is for thebidirectional RNN portion. In some embodiments, the parameters (V, W)can be jointly optimized by minimizing the objective function. In someembodiments, the objective function can be:

${L = {{- \frac{1}{D}}{\sum_{{({x_{t},y_{t}})}_{t = 1}^{T} \in D}{\sum\limits_{t = 1}^{T}{\log\;{P\left( {\left. y_{t} \middle| {x_{1:t}y_{1:{t - 1}}} \right.,V,W} \right)}}}}}},$

Where D represents the training data set, |D| represents the number ofsamples in the training set, T represents the length of each sample inthe training set, P represents the probability, back propagation can beused to calculate the gradient ∇_(V,W)L(V, W), and the stochasticgradient descent method is used to optimize the parameters (V, W).

Although the stochastic gradient descent method and the objectivefunction L are disclosed as examples, other objective functions may beemployed, including but not limited to cross entropy, etc., and otherparameter optimization methods, including but not limited to adaptivemoment estimation, etc., may also be employed. After confirming that alltraining data has been processed in step 505, a predictive network whoseparameters have been optimized on all training data may be output (step506). In some embodiments, the training of the predictive network mayalso be ended after the convergence of the objective function isconfirmed. In some embodiments, a regularization method of L1 or L2 canalso be used to avoid over-fitting. In some embodiments, the trainingdata set can also be divided into batches and trained using a smallbatch gradient descent method to reduce computational load whileavoiding over-fitting.

FIG. 6 shows a schematic illustration of a vessel stenosis degreeprediction system 600 in accordance with an embodiment of the presentdisclosure. As shown in FIG. 6, the left side of the dashed line is theoffline training process: the established prediction model istransmitted to the prediction model training unit 603, which trains theprediction model by using training samples from the training sampledatabase 604. Each training sample can be composed of image blocks at aseries of stenosis locations manually labeled by the physician on theblood vessel path on the vessel image and the degree of stenosis at thecorresponding locations. Compared to the independent training of thevarious modules in the existing prediction model, training samples usedfor such end-to-end training are easier to acquire and build. Thetrained prediction model is fed into a prediction unit 604 to calculatea predicted result based on the model input. In some embodiments, theblood vessel path corresponding to the training sample conforms to thesame or similar spatial relationship as the blood vessel path as thestenosis degree prediction target (e.g., the number of sampling pointsand the spatial constraint relationship between the sampling points),such that the trained prediction model can be seamlessly interfaced withthe predicted target without the need for conversion training and canfurther improve the accuracy of the prediction.

To the right of the dotted line is the online prediction process. First,a medical image of a patient is obtained from the medical image database606, and a CT image satisfying the DICOM standard of the blood vesseltree will be described below as an example. It is noted that the medicalimage database 606 can be a database of patient medical images sharedwithin the hospital or among the hospitals. The CT image of the bloodvessel tree can be fed into a reconstruction unit 601, which can bedeveloped, for example, using existing blood vessel reconstructionsoftware, or by using open source software such as VMTK, to extract thecenterline and blood vessel wall from the CT image of the blood vesseltree to reconstruct the geometric model of each blood vessel path(optionally reconstructing the blood vessel tree). The geometric modelof the reconstructed blood vessel path may be fed to the image blockintercepting unit 605 and the candidate stenosis range setting unit 602.The candidate stenosis range setting unit 602 includes a healthy radiusprediction unit 607 and a candidate stenosis prediction unit 608,wherein the healthy radius prediction unit 607 can extract a bloodvessel radius of the corresponding blood vessel path based on thereconstructed geometric model of each blood vessel path, and then thehealthy radius of the blood vessel path may be predicted by using amethod for predicting the healthy radius of the blood vessel pathaccording to various embodiments of the present disclosure, and thecandidate stenosis prediction unit 608 may obtain the blood vesselradius of the corresponding blood vessel path from the healthy radiusprediction unit 607 and detect the radius therefrom radius valley, andthe healthy radius of the predicted blood vessel path is then obtainedfrom the healthy radius prediction unit 607, thereby determining thecandidate stenosis by utilizing the method for predicting the candidatestenosis of the blood vessel path according to various embodiments ofthe present disclosure, based on the predicted healthy radius of theblood vessel path and the obtained radius valley of the blood vesselpath. In some embodiments, the candidate stenosis range setting unit 602may set a candidate stenosis range based on the determined candidatestenosis. For example, the candidate stenosis range may be set to alength including all points on both sides where blood vessel radii aresmaller than the corresponding healthy radius with the candidatestenosis point being a center.

The image block intercepting unit 605 may receive the set candidatestenosis range from the candidate stenosis range setting unit 602, andintercept a certain number of image blocks along the centerline of theblood vessel within the candidate stenosis range from the reconstructedgeometric model of the blood vessel path. The size of the image blockcan be preset according to user's needs and experience. In someembodiments, the prediction model is constructed based on aconvolutional neural network and a recurrent neural network, and thenumber of image blocks may be preset to be the same as the number ofnodes of the recurrent neural network, and the number of nodes of therecurrent neural network may be preset according to a user's needs andexperience.

A certain number of image blocks intercepted by the image blockintercepting unit 605 along the centerline may be fed into a predictionunit 604, which may be configured to predict the degree of stenosis ofthe corresponding blood vessel path using the trained prediction modelbased on the acquired image block sequence.

FIG. 7 shows a schematic block diagram of a blood vessel stenosis degreeprediction device 700 according to an embodiment of the presentdisclosure. The vessel stenosis degree prediction device 700 can includea network interface 708 via which the vessel stenosis degree predictiondevice 700 can be connected to a network (not shown), such as, but notlimited to, a local area network or the Internet in a hospital. Thenetwork may connect the blood vessel stenosis degree prediction device700 to an external device such as an image acquisition device (notshown), a medical image database 705, and an image data storage device706.

In some embodiments, anatomical tree structure analysis device 700 canbe a dedicated smart device or a general-purpose smart device. Forexample, the device 700 can be a computer customized for image dataacquisition and image data processing tasks, or a server located in thecloud. For example, device 700 can be integrated into an imageacquisition device. Alternatively, the (at least one) image processingprogram 703 in the device 700 may comprise or cooperate with a 3Dreconstruction unit for reconstructing a 3D model of the blood vesselbased on the 2D blood vessel image acquired by the image acquisitiondevice, and the image blocks may be extracted from the 3D model as amodel input at a set of centerline points.

The blood vessel stenosis degree prediction device 700 may include animage processor 701 and a memory 702, and may additionally include atleast one of an input/output 707 and an image display 709.

The image processor 701 can be any of the processors described invarious embodiments of the present disclosure. The disclosed embodimentsare not limited to any type of processor or processor circuit that isotherwise configured to satisfy the computational requirements foridentifying, analyzing, maintaining, generating, and/or providing alarge amount of imaging data or manipulating such imaging data ormanipulating any other type of data in consistent with the disclosedembodiments. Additionally, the term “processor” or “image processor” mayinclude more than one processor, such as a multi-core design or multipleprocessors, each having a multi-core design. The image processor 701 canexecute a sequence of computer program instructions stored in the memory702 to perform the various operations, processes, and methods disclosedherein.

The image processor 701 can be communicatively coupled to the memory 702and configured to execute computer executable instructions storedtherein. The memory 702 may include read only memory (ROM), flashmemory, random access memory (RAM), dynamic random access memory (DRAM)such as synchronous DRAM (SDRAM) or Rambus DRAM, static memory (e.g.flash memory, static random access memory) and the like, and computerexecutable instructions are stored therein in any format. In someembodiments, the memory 702 can store computer-executable instructionsof one or more image processing programs 703. Computer programinstructions may be accessed by image processor 701, read from ROM orany other suitable memory location, and loaded into RAM for execution byimage processor 701. For example, the memory 702 can store one or moresoftware applications. The software applications stored in the memory702 may include, for example, an operating system (not shown) for ageneral computer system and an operating system for the soft controldevice.

Moreover, the memory 702 can store the entire software application oronly a portion of the software application (e.g. image processingprogram 703) that can be executed by the image processor 701. Inaddition, the memory 702 can store a plurality of software modules forimplementing various steps of various methods consistent with thepresent disclosure for predicting a healthy radius of a blood vesselpath, predicting a candidate stenosis of a blood vessel path, andpredicting a degree of stenosis of a blood vessel path. For example, theprediction model training unit 603, the prediction unit 604, the imageblock intercepting unit 605, the reconstruction unit 601, and thecandidate stenosis range setting unit 602 (as shown in FIG. 6) may beimplemented as software modules stored on the memory 702. As anotherexample, at least the prediction unit 604, the image block interceptingunit 605, the reconstruction unit 601, and the candidate stenosis rangesetting unit 602 may be implemented as a software module stored in thememory 702. The prediction model training unit 603 may be located awayfrom the blood vessel stenosis degree prediction device 700, and can becommunicated with the prediction unit 604 to enable it to receive anupdated prediction model that can be trained by prediction modeltraining unit 603 using training samples from training sample database604 (in an offline training process) and/or training samples from theprediction unit 604 (i.e., the stenosis degree prediction results alongwith the image blocks extracted at the corresponding locations) (in theonline training process).

Moreover, memory 702 can store data generated/cached when the computerprogram is executed, for example, medical image data 704, includingmedical images transmitted from image acquisition device, medical imagedatabase 705, image data storage device 706, and the like. In someembodiments, the medical image data 704 can include an image of thereceived vessel tree, and the (at least one) image processing program703 implements centerline extraction and 3D model reconstruction, imageblock extraction (as a model input) and prediction of degree of bloodvessel stenosis for the image of the vessel tree. In some embodiments,the medical image data 704 can include a volumetric image of thereceived vessel tree, and the (at least one) image processing program703 is to perform an image block extraction along the centerline for thevolumetric image of the vessel tree (as a model input) and prediction ofdegree of blood vessel stenosis. In some embodiments, the memory 702 canload a batch of training samples from the medical image database 705 andtemporarily store it as medical image data 704 for use by predictionmodel training unit 603 for small batch training. In some embodiments,memory 702 can temporarily store prediction results and correspondingmodel inputs as online training samples. The training samples stored asthe medical image data 704 can be deleted after the training with themis completed so as to free up the space of the memory 702 and increaseits capacity and performance.

In some embodiments, the prediction model can be stored in the medicalimage data 704, and be used in the next prediction of degree of bloodvessel stenosis (used after training). In some embodiments, the updatedand optimized parameters of the trained prediction model can be storedin the medical image data 704, and can be utilized in the nextprediction of degree of blood vessel stenosis of the corresponding bloodvessel path of the same patient.

In some embodiments, the image processor 701, after predicting thestenosis distribution of the blood vessel path, can associate the imageof the blood vessel tree with the predicted result as medical image data704 for presentation and/or transmission. In some embodiments, an imageof the vessel tree along with the predicted results can be displayed onimage display 709 for viewing by the user. For example, the imagedisplay 709 can be an LCD, CRT, or LED display. The input/output 707 mayprovide medical image data 704 to the display 709. In this way, the usercan utilize input/output 707 to confirm and correct the displayedpredictions, if desired. The confirmed and corrected blood vesselstenosis prediction result may be temporarily stored in the memory 702as medical image data 704 in association with a model input such as animage block along the center line, and may be transmitted to the medicalimage database 705 for being accessed, obtained and used by anothermedical device (such as other anatomical tree structure analysis device700) as needed.

In some embodiments, the memory 702 can be in communication with themedical image database 705 to transmit and save the extracted modelinput into the medical image database 705 in association with theprediction results as a piece of training data that can be used foroffline training. In this manner, the training sample database 604 asshown in FIG. 6 can be incorporated into the medical image database 705.

In addition, the parameters of the trained prediction model can bestored in the medical image database 705 for access and acquisition. Theparameters of the trained prediction model can be used by other vesselstenosis degree prediction devices 700, if desired.

In some embodiments, the medical image database 606 shown in FIG. 6 canbe included into the medical image database 705, which can maintainmedical images and/or 3D models and/or centerlines and/or a series ofimage blocks along the centerline of the vessel tree according to thepatient. Thus, the memory 702 can communicate with the medical imagedatabase 705 to obtain at least one of a medical image, a 3D model, acenterline, and a series of image blocks along the centerline of thecurrent user's vessel tree. The 3D model, the centerline, and a seriesof image blocks along the centerline of the vessel tree can beconstructed and extracted by the reconstruction unit 601 shown in FIG.6, and transmitted to the medical image database 705 for storage by theuser.

In some embodiments, image data storage device 706 can be provided toexchange image data with the medical image database 705. For example,the image data storage device 706 can reside in other medical imageacquisition devices, such as a CT that performs a volumetric scan of apatient. The patient's volumetric image can be transmitted and savedinto the medical image database 705, and the vessel stenosis degreeprediction device 700 can take a volumetric image and a prediction modelof the particular patient from the medical image database 705 and make aprediction of degree of stenosis based thereon.

The input/output 707 can be configured to allow blood vessel stenosisdegree prediction device 700 to receive and/or transmit data.Input/output 707 may include one or more digital and/or analogcommunication devices that allow device 700 to communicate with a useror other machine(s) and device(s). For example, input/output 707 caninclude a keyboard and mouse that allow a user to provide input.

The network interface 708 may include a network adapter, a cableconnector, a serial connector, a USB connector, a parallel connector, ahigh speed data transfer adapter such as fiber optics, USB 6.0,lightning, a wireless network adapter such as a Wi-Fi adapter,telecommunications (6G, 4G/LTE, etc.) adapters. Device 700 can beconnected to the network via network interface 708. The network mayprovide functions such as a local area network (LAN), a wirelessnetwork, a cloud computing environment (e.g. software as a service, aplatform as a service, an infrastructure as a service, etc.), aclient-server, a wide area network (WAN), and the like.

The various steps, operations, or functions described above may beimplemented or defined as software code or instructions. Such contentmay be directly executable (“object” or “executable” form), source codeor differential code (“delta” or “patch” code). The softwareimplementation of the embodiments described herein may be provided viaan article having code or instructions stored thereon or via a method ofoperating a communication interface to transmit data via a communicationinterface. For example, a non-transitory computer-readable medium canencode instructions that, when executed in hardware, perform a process,including any of the various steps, operations, or functions describedabove.

Execution of the order or operation of the embodiments of the presentinvention shown and described herein is not essential, unless otherwiseindicated. That is, operations may be executed in any order, unlessotherwise stated, and embodiments of the invention may include more orfewer operations than those disclosed herein. For example, executing orimplementing a particular operation before, concurrently with, or afteranother operation is contemplated to be within the scope of theinventive arrangements.

Embodiments of the invention may be implemented using computerexecutable instructions. Computer executable instructions can beorganized into one or more computer executable components or modules.The various aspects of the invention can be implemented using any numberand organization of such components or modules. For example, aspects ofthe invention are not limited to the specific computer-executableinstructions or specific components or modules shown in the drawings anddescribed herein. Other embodiments of the invention may includedifferent computer-executable instructions or components having more orless functionality than those illustrated and described herein.

The various aspects of the present invention have been described indetail, and it is obvious that modifications and variations are possiblewithout departing from the scope of the invention as defined in theappended claims. Since various changes to the above-describedstructures, products, and methods can be made without departing from thescope of the present invention, it is intended that all of the contentsincluded in the above description and shown in the drawings should beconstrued as exemplary without restrictive meaning.

What is claimed is:
 1. A computer-implemented method for predicting ablood vessel stenosis, the method comprising: extracting a blood vesselpath and its centerline based on an image of a blood vessel; determininga candidate stenosis for the blood vessel path; identifying image blocksalong the centerline of the blood vessel path within a range ofcandidate stenosis for the blood vessel path determined based on thecandidate stenosis; and determining a degree of stenosis for the bloodvessel path by applying a trained learning network comprising aconvolutional neural network and a recurrent neural network on the imageblocks within the range of candidate stenosis.
 2. Thecomputer-implemented method according to claim 1, further comprising:predicting a healthy radius of the blood vessel path by performing aquadratic regression on blood vessel radius peaks of the blood vesselpath; and determining the candidate stenosis for the blood vessel pathbased on the healthy radius of the blood vessel path.
 3. Thecomputer-implemented method according to claim 2, wherein predicting thehealthy radius of the blood vessel path further comprises: predicting areference healthy radius of the blood vessel path by performing a linearregression on the blood vessel radius peaks; and replacing the bloodvessel radius peaks in the blood vessel path that are lower than thereference healthy radius with the reference healthy radius.
 4. Thecomputer-implemented method according to claim 1, wherein the range ofcandidate stenosis for the blood vessel path has the candidate stenosisat its center.
 5. The computer-implemented method according to claim 1,wherein the convolutional neural network and the recurrent neuralnetwork are sequentially applied, wherein the convolutional neuralnetwork is applied on the image blocks along the centerline of the bloodvessel path to generate vectors for the respective image blocks, whereinthe recurrent neural network is applied on the vectors.
 6. Thecomputer-implemented method according to claim 1, wherein theconvolutional neural network and the recurrent neural network arejointly trained.
 7. The computer-implemented method according to claim1, wherein the convolutional neural network comprises a plurality ofconvolutional neural network units each configured to process one of theimage blocks along the centerline of the blood vessel path.
 8. Thecomputer-implemented method according to claim 1, wherein the recurrentneural network comprises a forward recurrent neural network and areverse recurrent neural network configured to collectively learn aspatial correlation among the image blocks along the centerline of theblood vessel path.
 9. The computer-implemented method according to claim8, wherein the forward recurrent neural network and the reverserecurrent neural network are separately trained.
 10. Thecomputer-implemented method according to claim 1, further comprisingoutputting at least one of the candidate stenosis, the range ofcandidate stenosis, and the degree of stenosis of the blood vessel path.11. A blood vessel stenosis degree prediction system, comprising: aninterface configured to receive an image of a blood vessel; a processorconfigured to: extract a blood vessel path and its centerline based onthe image of the blood vessel; determine a candidate stenosis for theblood vessel path; identify image blocks along the centerline of theblood vessel path within a range of candidate stenosis for the bloodvessel path determined based on the candidate stenosis; and determine adegree of stenosis for the blood vessel path by applying a trainedlearning network comprising a convolutional neural network and arecurrent neural network on the image blocks within the range ofcandidate stenosis.
 12. The blood vessel stenosis degree predictionsystem according to claim 11, wherein the processor is furtherconfigured to: predict a healthy radius of the blood vessel path byperforming a quadratic regression on blood vessel radius peaks of theblood vessel path; and determine the candidate stenosis for the bloodvessel path based on the healthy radius of the blood vessel path. 13.The blood vessel stenosis degree prediction system according to claim11, wherein to predict the healthy radius of the blood vessel path, theprocessor is further configured to: predict a reference healthy radiusof the blood vessel path by performing a linear regression on the bloodvessel radius peaks; and replace the blood vessel radius peaks in theblood vessel path that are lower than the reference healthy radius withthe reference healthy radius.
 14. The blood vessel stenosis degreeprediction system according to claim 11, wherein the range of candidatestenosis for the blood vessel path has the candidate stenosis at itscenter.
 15. The blood vessel stenosis degree prediction system accordingto claim 11, wherein to determine the degree of stenosis for the bloodvessel path, the processor is configured to apply the convolutionalneural network and the recurrent neural network sequentially, whereinthe convolutional neural network is applied on the image blocks alongthe centerline of the blood vessel path to generate vectors for therespective image blocks, wherein the recurrent neural network is appliedon the vectors.
 16. The blood vessel stenosis degree prediction systemaccording to claim 11, wherein the convolutional neural network and therecurrent neural network are jointly trained.
 17. The blood vesselstenosis degree prediction system according to claim 11, wherein therecurrent neural network comprises a forward recurrent neural networkand a reverse recurrent neural network configured to collectively learna spatial correlation among the image blocks along the centerline of theblood vessel path.
 18. The blood vessel stenosis degree predictionsystem according to claim 11, further comprising a display configured tooutput at least one of the candidate stenosis, the range of candidatestenosis, and the degree of stenosis of the blood vessel path.
 19. Acomputer-implemented method for training a learning network forpredicting a blood vessel stenosis, the learning network comprising aconvolutional neural network and a recurrent neural network, comprising:receiving training data including image blocks at a plurality ofstenosis locations along a blood vessel path and a labeled degree ofstenosis for the stenosis locations; determining a predicted degree ofstenosis by applying the learning network to the image blocks in thetraining data; jointly optimizing a first set of parameters of theconvolutional neural network and a second set of parameters of therecurrent neural network against an objective function determined basedon the labeled degree of stenosis in the training data and the predicteddegree of stenosis; and providing the learning network updated with thejointly optimized first set of parameters and second set of parameters.20. The computer-implemented method according to claim 19, wherein therecurrent neural network comprises a forward recurrent neural networkand a reverse recurrent neural network configured to collectively learna spatial correlation among the image blocks at the plurality ofstenosis locations along the blood vessel path.